Consistent Classification Algorithms for Multi-class Non-Decomposable Performance Metrics

We study consistency of learning algorithms for a multi-class performance metric that is anon-decomposable function of the confusion matrix of a classifier and cannot be expressed asa sum of losses on individual data points; examples of such performance metrics include themicro and macro F-measure used widely in information retrieval and the multi-class G-meanmetric popular in class-imbalanced problems. While there has been much work in recent years inunderstanding the consistency properties of learning algorithms for ‘binary’ non-decomposablemetrics [1, 2, 3, 4, 5], little is known either about the form of the optimal classifier for a generalmulti-class non-decomposable metric, or about how these learning algorithms generalize to themulti-class case. In this paper, we provide a unified framework for analyzing a multi-class non-decomposable performance metric, where the problem of finding the optimal classifier for theperformance metric is viewed as an optimization problem over the space of all confusion matricesachievable under the given distribution. Using this framework, we show that (under a continu-ous distribution) the optimal classifier for a multi-class performance metric can be obtained asthe solution of a cost-sensitive classification problem, thus generalizing several previous resultson specific binary non-decomposable metrics. We then design a consistent learning algorithmfor concave multi-class performance metrics that proceeds via a sequence of cost-sensitive clas-sification problems, and can be seen as applying the conditional gradient (CG) optimizationmethod over the space of feasible confusion matrices. To our knowledge, this is the first efficientlearning algorithm (whose running time is polynomial in the number of classes) that is provablyconsistent for a large family of multi-class non-decomposable metrics. Our consistency resultmakes use of a novel proof technique based on the convergence analysis of the CG method.